[Ed. Note: We happen to have a hole in the schedule today; so I’m repurposing a relevant post from elsewhere for SBM, because it’s a study that was in the news a lot and is now making the rounds on antivax social media. It’s been spiffed up a bit for SBM.
Also: If you’ve come here from the NYT article citing this one, please read my whole post. I think that the NYT somewhat misrepresented my actual position here.]
Earlier this week, a CDC-funded study of the flu vaccine hit the news, and the antivaccine movement went wild. The reason was due to headlines like this, including in some major news outlets:
- “Study Linking Early Miscarriage to Flu Vaccine Puzzles Doctors“
- “Study shows miscarriage risk may have increased after flu shots, puzzling researchers“
- “What to know about a study of flu vaccine and miscarriage“
- “Study signals association between flu vaccine, miscarriage“
In the antivaccine world, the blog post titles were practically giddy in their fear mongering, such as “Devastating flu vaccine-miscarriage study sparks ridiculous spin” (J.B. Handley).
The crankiest of these cranks, of course, is Mike Adams, a grifter deep in the thrall of any form of pseudoscience that he can sell to burnish his brand and keep the rubes buying and who knows how to whip his minions into a fine frothy head of anti-pharma conspiracy mongerings:
A CDC-funded medical study being published by the medical journal Vaccine has confirmed a shocking link between flu shots and spontaneous abortions in pregnant women. The study was rejected by two previous medical journals before Vaccine agreed to publish it, further underscoring the tendency for medical journals to censor any science that doesn’t agree with their pro-vaccine narratives.
“A study published today in Vaccine suggests a strong association between receiving repeated doses of the seasonal influenza vaccine and miscarriage,” writes CIDRAP, the Center for Infectious Disease Research and Policy.
“A puzzling study of U.S. pregnancies found that women who had miscarriages between 2010 and 2012 were more likely to have had back-to-back annual flu shots that included protection against swine flu,” reports Medical Xpress, a pro-vaccine news site that promotes vaccine industry interests. Notice that the opening paragraph of their study assumed the study couldn’t possibly be true. It’s “puzzling” that mercury in flu shots could cause spontaneous abortions, you see, because these people have no understanding of biochemistry and the laws of cause and effect.
Actually, as has been documented so many times before, it is Mike Adams who has no understanding of biochemistry—or any other science—other than what it takes for him to portray himself to his gullible followers as a “real scientist.” As for the “laws of cause and effect,” whenever someone says something like that in reference to an epidemiological study, I know he’s really, really clueless, because if there’s anything that’s very difficult to do in an epidemiological study with reliability it’s determining cause-and-effect. That’s why the cardinal rule of epidemiology is that correlation does not equal causation. It might, but usually it doesn’t, and it usually takes a whole lot more than just one study with a correlation to start to suggest causation. This is particularly true when a study like the one Adams is gloating about is such an outlier, which this study most definitely is, as you will see. It’s also an exercise in data dredging that illustrates the danger of doing subgroup analyses in which each subgroup has small numbers.
Let’s go to the study itself. I can’t help but note that Frank DeStefano of the CDC is a co-author. DeStefano, as you might recall, is one of those CDC investigators that antivax conspiracy theorists like those who made the propaganda film VAXXED portray as one of the main villains in the “CDC whistleblower” conspiracy theory. Also, several of the authors receive pharma money for research support. Nicola Klein, for instance, receives research support from GlaxoSmithKline, Sanofi Pasteur, Pfizer, Merck, MedImmune, Novartis, and Protein Science, while Allison Naleway receives funding from GlaxoSmithKline, MedImmune, and Pfizer. Others receive support from MedImmune and Novavax. So basically, this was a study funded by the CDC and carried out by CDC scientists and scientists receiving significant pharma funding. I just couldn’t resist pointing that out. I know, I know, antivaxers will claim that the findings were so compelling that not even the CDC and pharma shills could hide them, but it would amuse me to point these things out to antivaxers.
Yet, here we see Del Bigtree, producer of VAXXED, gleefully citing J.B. Handley gloating over this study:
Tough breaking news for @ChelseaClinton @DrPanMD @doritmi FLU SHOT linked to MISCARRIAGE! #endvaxinjury https://t.co/6JwjnsAviY
— Del Bigtree (@delbigtree) September 13, 2017
I wonder if Bigtree knows that DeStefano is a co-author. He probably doesn’t care, because DeStefano, like any scientist, can be a hero or a villain depending solely upon whether he produces information or studies that agrees with the antivaccine narrative that the flu vaccine is not just useless but dangerous. Be that as it may, the article above is a typical bit of Handley’s Dunning-Krugger arrogance of ignorance, even more full of hyperbole and nonsense than the usual Mike Adams’ endeavors in that area. Indeed, Handley even uses the argumentum ad package insert gambit. (Whenever I see that gambit used by an antivaxer, my estimation of his cluelessness goes up several notches, which for Handley is really saying something.) In any case, you can get a feel for how much the authors of this study are stretching to find a correlation—any correlation—between influenza vaccines and miscarriages that they’re looking at combinations of vaccines by their brief justification for the study:
Since 2004, the Centers for Disease Control and Prevention (CDC) Advisory Committee on Immunization Practices (ACIP) and other organizations have recommended routine influenza vaccination for pregnant women regardless of gestational age [1,2]. Influenza in pregnancy can cause serious, life-threatening illness in both the mother and fetus, as demonstrated during the 2009 pandemic [3,4]. Numerous studies of influenza vaccine during pregnancy have not identified serious safety concerns [5–12], but relatively few investigations have evaluated vaccination in the first trimester, a period when the embryo is highly vulnerable to teratogens and other factors [5,13]. A case-control study conducted by the Vaccine Safety Datalink (VSD) demonstrated that influenza vaccination during early pregnancy in the 2005–06 and 2006–07 influenza seasons was not associated with spontaneous abortion (SAB) [14].
The emergence of a pandemic influenza virus, A/California/ 7/2009 (H1N1)pdm09 (pH1N1), led to rapid development and widespread use of vaccines containing pH1N1 antigens. Several studies have evaluated the safety of vaccines containing pH1N1 in pregnancy, but few have focused on outcomes in early pregnancy [15–19]. Using a design and protocol similar to the previous study [14], we conducted a case-control study to determine if receipt of influenza vaccine containing pH1N1 was associated with SAB.
Notice the eight studies cited (references 5-12) that failed to find significant safety issues with the vaccine in pregnancy, and a study (reference 14) using VSD data failed to find an association between flu vaccination with spontaneous abortion. That’s actually a lot of data for the safety of the flu vaccine during pregnancy, which makes me wonder what the justification for yet another study looking for an association between influenza vaccination and miscarriages. If I were a funding agency and received a grant application to do a study like this with text above in the “Background and Significance” or the “Impact” section, my first reaction would be: Why on earth would we fund this? It’s all been done before, many, many times. Yet the CDC funded this study. So much for antivax claims about the CDC not being concerned about vaccine safety and not being willing to look for adverse reactions due to vaccines.
I also find it rather odd that the authors would say that few studies have been done looking for a correlation between vaccination against influenza, even in early pregnancy, when in fact there have been a lot, many well-designed, and they’ve pretty much all been negative. Whenever you see a study that finds something a lot different from the bulk of the studies that have been done before, the first question to be asked is: Are the results of the current study so robust that they indicate a hole in the existing data addressing the question asked that we should begin to question the cumulative results of all the studies that have gone before? Keep that question in mind as I continue.
Also consider the bias that exists in journals to publish novel findings. As this news report points out, this is the “first study to identify a potential link between miscarriage and the flu vaccine.” That’s almost certainly the reason that it was published. Adams, in his haste to portray as a conspiracy to silence, inadvertently tells me something. That this paper was rejected by two previous journals is not surprising to me. What is surprising is that Vaccine ultimately accepted it, which was reported in the AP version of this story. I’m not sure why Gregory Poland, the editor of Vaccine, felt obligated to point that out to a reporter. One would think that a journal editor wouldn’t want to advertise that he accepted a manuscript that had been rejected elsewhere. My guess is that the authors submitted it to epidemiology journals, where reviewers dinged it for its obvious flaws, but I have no way of knowing.
So what about the study itself? First, it’s a case-control study. Basically, that means that the authors studied a cohort of women who had miscarriages (the cases) and compared them to a cohort of women who didn’t have miscarriages but instead delivered full term infants or had stillbirths during the study period (the controls). The authors chose two flu seasons (2010 to 2012) and asked if women who had miscarriages were more likely to have been vaccinated for influenza within 28 days prior to miscarriage, as well as for different time periods before miscarriage. That’s how case control studies work. You compare patients who have a medical condition (the cases, in this case, miscarriages) to those who do not (the controls), matching them as closely as possible in other relevant traits, and then ask whether there is a difference in the exposure under study between the groups.
The most critical aspect of any case control study if, of course, the matching of cases to controls. The idea is to match them as closely as possible on all relevant factors other than the condition under investigation (in this case, miscarriage). Not uncommonly, investigators will do a 2:1 match, controls to cases, in order to make the comparison more robust. It’s not mandatory, and Donohue et al chose not to do this. In this study, cases had SAB and controls had live births or stillbirths and were matched on site, date of last menstrual period, and age. I also note that the database they used was the Vaccine Safety Datalink (VSD). As I like to say, the VSD is an excellent rebuke to antivaxers who claim that doctors don’t care about vaccine safety. It’s a database designed to document adverse events associated with vaccination, and it’s a huge database.
So what did the study find? Here’s a summary of the cases analyzed:
I can’t help but note that the matching leaves something to be desired. Cases and controls were matched to last menstrual period, which is appropriate to assure nearly identical gestational ages. Other matching included VSD site, maternal age (30 years was the cutoff). That’s some rather loose matching, in my book, given that there are a lot of other potential factors that can increase the risk of spontaneous abortion (miscarriage). More on this later.
If you look at the tables in the paper, the first thing you will notice is that the adjusted odds ratios (aORs) for miscarriage as a function of having received the flu vaccine are nearly all around 1.0 or not statistically different from 1.0. Obviously, there are exceptions. Basically, the study found that, if a woman had consecutively received a flu vaccine containing the 2009 H1N1 virus the season before and had the flu vaccine in one of the two seasons studied, the aOR in the 1–28 days was 7.7 (95% CI 2.2–27.3). Otherwise, the aOR was 1.3 (95% CI 0.7–2.7) among women not vaccinated in the previous season; i.e., not statistically significant from 1.0, meaning no detectable difference in miscarriage rates compared to women who had not been vaccinated. This effect was noted in both seasons.
Now here’s where you should be skeptical.
This study has several important limitations. First, the most striking findings relate to the association between SAB and IIV [inactivated influenza vaccine] in women who previously received pH1N1-containing vaccine. This interaction effect was not an a priori hypothesis; the results were generated in a post hoc analysis with small numbers of women in the various subgroups. Although the interaction was observed in each of the two seasons studied, the point estimates were substantially larger (though not statistically different) in the first season for reasons that are unclear. Second, although most cases had an ultrasound, assignment of a precise date of SAB was challenging. With guidance from an obstetrician we integrated different types of information from the medical record (e.g., ultrasound results, clinical and laboratory findings, provider notes) to estimate the timing of the SAB. Estimation of SAB dates was independent of vaccination status so any error should bias the results toward the null (i.e., non-differential misclassification). Third, we studied only women who had clinically confirmed SAB; the proportion of women with clinically unrecognized pregnancy loss is uncertain but may be substantial [50,51]. Our results could be biased if women who sought care for SAB were more likely to be vaccinated in the 28-day exposure window.
So what we’re looking at is an association, nothing more. It’s an association with a lot of caveats, too. Basically, having found nothing more than one association with an aOR of 2.0 for the 1-28 day window of exposure to the influenza vaccine before miscarriage that was barely statistically significant (95% confidence interval: 1.1-3.6), the authors did a post hoc analysis looking for other associations. (Never mind that the “association” they found was eminently unimpressive given the size of the confidence intervals.) “Post hoc” means that they did additional analyses not originally specified. Basically investigators don’t usually do post hoc analyses if there is a robust association in their data. They do it when they fail to find an association or only find an unimpressive association that is not robust. Also, post hoc analyses are prone to type 1 errors, which means finding a statistically significant “association” where there is none; i.e., finding a false positive. When the numbers in the subgroup are so small and the study is observational (i.e., retrospective), that tendency is even stronger. Then there was the issue that the cases and controls were not as comparable as one would like in a case control study. For example, cases were significantly older than controls and more likely to be African-American, to have a history of 2 spontaneous abortions, and to have smoked during pregnancy. The authors did some correcting for age and history of spontaneous abortions, but it’s questionable to me whether it was adequate.
Basically, the authors did what we refer to as a subgroup analysis, in this case the subgroup being women who had received H1N1 vaccination the season before they received the flu vaccination in the seasons examined? That clearly wasn’t a primary hypothesis being tested. Rather, it was a hypothesis the authors clearly came up with while doing the study. One wonders if this analysis was prespecified or whether the protocol was changed midway through. I only ask that because antivaxers went wild over claims by the “CDC whistleblower” that the Atlanta MMR study changed its protocol part way through the study, but, here, where the analysis seems to suggest such an “adjustment” during the study (although it is certainly possible that the H1N1 analysis was prespecified in the original protocol, given the choice of the 2010-11 and 2011-12 flu seasons), we hear…silence. Whatever the case, there clearly was post hoc analysis going on, so much so that I smell= p-hacking.
Speculations about the protocol aside, the investigators found what they found, namely an aOR of 7.7 for cases versus controls for exposure to the H1N1 vaccine the year before plus the flu vaccine within 1-28 days before their miscarriages. This was based on some very small numbers, though, namely 14 miscarriages and 4 controls. In other words, this is almost certainly a statistical fluke, given that it was only found for women who had received H1N1 the season before and had received the flu vaccine within 28 days of their miscarriage, and that the association was not observed for pretty much any other time window or combination. When considering such a result, one also has to consider biological mechanism and plausibility, and it is just not very plausible from a biological or immunological standpoint that this combination of flu vaccines—and only this combination—given only during a specific time window will cause miscarriages. Like Dr. Gregory Poland, the editor of Vaccine, I don’t believe these findings, either. Unlike Dr. Poland, I wouldn’t have published these findings, given how very likely it is that they are a statistical fluke.
I particularly don’t believe them in light of what we already know, based on studies Tara Haelle summarized the data with respect to flu vaccines and miscarriages, stillbirths, and birth defects in 2014, using mainly studies published during the prior two years, and the results were very consistent and overwhelming: There was no association between vaccination for influenza and adverse fetal outcomes. Just for yucks, I did some PubMed searches myself for more recent studies, and found basically the same thing, but instead of listing those studies, I’ll just refer you to a recent large meta-analysis that found that the risk of stillbirth was actually lower in women vaccinated against influenza and no difference in the risk of spontaneous abortion. In other words, this new study is an outlier. It’s such an outlier, that scientists are correct to be very skeptical of its results. Heck, even the authors are skeptical of its results. Unfortunately, they’re not so skeptical that they don’t resist making the call for “more research.” They’ll probably get the funding for that “more research,” and then when the inevitable negative study is finally published, no one will remember it. They’ll all remember this study, and, of course, the antivaccine movement will be flogging it for years to come, beginning now.